An elastic framework for ensemble-based large-scale data assimilation

Prediction of chaotic systems relies on a floating fusion sensor data (observations) with numerical model to decide good system trajectory and compensate non-linear feedback effects. Ensemble-based assimilation (DA) is major method for this concern depending propagating an ensemble perturbed realizations. In paper, we develop elastic, online, fault-tolerant modular framework called Melissa-DA large-scale ensemble-based DA. Melissa-DAallows elastic addition or removal compute resources state propagation at runtime. Dynamic load balancing based list scheduling ensures efficient execution. Online processing the produced by members enables avoid I/O bottleneck file-based approaches. Our implementation embeds PDAF parallel DA engine, enabling use various methods. Melissa-DAcan support extra methods implementing transformation member background states into analysis states. Experiments confirm excellent scalability Melissa-DA, 16,384 regional hydrological critical zone relying ParFlow domain about 4 M grid cells. The same case was ported state-of-the-art MPI approach. A comparison 2500 20,000 cores shows our approach 50% faster per cycle.